Classifying a lesion based on longitudinal studies

ABSTRACT

A computer-implemented method is for classifying a lesion. In an embodiment, the method includes receiving a first medical image of an examination volume, the first medical image corresponding to a first examination time; receiving a second medical image of the examination volume, the second medical image corresponding to a second examination time, different from the first examination time; determining a first lesion area corresponding to a lesion within the first medical image; determining a registration function based on a comparison of the first medical image and the second medical image; determining a second lesion area within the second medical image based on the registration function and the first lesion area; and classifying the lesion within the first medical image based on the second lesion area. A computer-implemented method for providing a trained classification function, a classification system, and computer program products and computer-readable media are also disclosed.

PRIORITY STATEMENT

The present application hereby claims priority under 35 U.S.C. § 119 toEuropean patent application number EP 20171081.1 filed Apr. 23, 2020,the entire contents of which are hereby incorporated herein byreference.

FIELD

Example embodiments of the invention generally relate to a method andapparatus for classifying a lesion.

BACKGROUND

Patients with certain risk factors should undergo regular screeningtests. For example, for heavy smokers between 55 and 80 years a low-dosecomputed tomography (an acronym is “CT”) based lung cancer screening isrecommended.

Based on the findings, typically lung nodules between 6 mm to 30 mm,detected in each screening, follow-up screening tests (also denoted as“longitudinal studies”) are scheduled to confirm the malignancypredictions before patients are being sent to pathologic evaluation. Dueto the small size of lung nodules, detecting lung nodules manually byradiologists is a time-consuming task. Besides the difficulty ofdetecting nodules in a single CT scan, it is also challenging forradiologists to track and compare multiple nodules between longitudinalCT scans to grade the malignancy of lung nodules. Though there aredifferent guidelines for guiding radiologists to make decisions forscheduling future follow-ups as well as calling patients for pathologytest, such decisions remain highly subjective.

From the paper Yang, Jie et al. “Class-Aware Adversarial Lung NoduleSynthesis in CT Images.” ISBI 2019, IEEE, 2019 it is known to handle thenodule malignancy grading as a 3D image patch classification problem.The nodules candidates are detected either by radiologists orcomputer-aided diagnosis systems. 3D image patches centered at thesecandidates are extracted from the CT images and sent to a deepconvolutional neural network-based classifier. The malignancyprobability of each nodule is then used for deriving the diagnosticdecisions. Some related methods also use the feature extracted withimage processing-based techniques to compensate neural networks. Inalternative methods, it is proposed to train auxiliary tasks, such asnodule type classification, jointly with the malignancy classificationtask to marginally improve the accuracy.

SUMMARY

Though the differential information between the longitudinal studies isthe major source of evidences that radiologists use for decision making,the inventors have discovered that these previous works are not capableof using such information.

At least one embodiment of the present invention is directed toimproving upon the diagnosis and characterization of lesions inlongitudinal medical images. The problem is improved upon or even solvedby a method for classifying a lesion, a method for providing a trainedclassification function, a classification system, a medical imaging, acomputer-program product and a computer-readable. Advantageousembodiments are described within the claims and within the followingdescription.

In the following, embodiments according to the invention are describedwith respect to systems as well as with respect to methods. Features,advantages or alternative embodiments herein can be assigned to theother corresponding objects and vice versa. In other words, the systemscan be improved with features described or claimed in the context of thecorresponding method. In this case, the functional features of themethods are embodied by objective units of the systems.

Furthermore, at least one embodiment according to the invention isdescribed with respect to methods and systems for providing a medicaldata record, with respect to methods and systems for classifying alesion, and with respect to methods and systems for providing a trainedclassifying function. Features, advantages or alternative embodimentsherein can be assigned to the other corresponding claimed objects andvice versa. In other words, features of the methods and systems forclassifying a lesion can be improved with features described or claimedin the context of methods and systems for providing a trainedclassifying function. In particular, the trained classifying functionprovided by a method or a system for providing the trained classifyingfunction can be used within a method or a system for classifying alesion.

According to a first embodiment, the invention relates to acomputer-implemented method for classifying a lesion, comprisingreceiving a first medical image of an examination volume, wherein thefirst medical image corresponds to a first examination time, furthermorecomprising receiving a second medical image of the examination volume,wherein the second medical image corresponds to a second examinationtime being different from the first examination time, furthermorecomprising determining a first lesion area corresponding to the lesionwithin the first medical image, furthermore comprising determining aregistration function based on a comparison of the first medical imageand the second medical image, determining a second lesion area withinthe second medical image based on the registration function and thefirst lesion area, and furthermore comprising classifying the lesionwithin the first medical image based on the second lesion area.

According to a second embodiment, the invention relates to acomputer-implemented method for providing a trained classifyingfunction. The method is based on receiving a first medical trainingimage of a training examination volume, wherein the first medicaltraining image corresponds to a first examination time, and on receivinga second medical training image of the training examination volume,wherein the second medical training image corresponds to a secondexamination time being different from the first examination time.Furthermore, the method is based on determining a registration functionbased on a comparison of the first medical training image and the secondmedical training image. Furthermore, the method is based on determininga first lesion area corresponding to a lesion within the first medicaltraining image. Furthermore, the method is based on receiving a trainingclassification corresponding to a first lesion area within the firstmedical training image. Furthermore, the method is based on determininga second lesion area within the second medical training image based onthe registration function and the first medical training image.Furthermore, the method is based on applying a trained classifyingfunction to first training input data and second training input data,thereby generating training output data, wherein the first traininginput data is based on the first lesion area, and wherein the secondtraining input data (TID.2) is based on the second lesion area (LA.2).Furthermore, the method is based on adjusting at least one parameter ofthe trained classifying function based on a comparison of the trainingclassification and the output data. Furthermore, the method is based onproviding the trained classifying function.

According to a third embodiment, the invention relates to aclassification system comprising

-   -   an interface configured for receiving a first medical image of        an examination volume, wherein the first medical image        corresponds to a first examination time,        furthermore configured for receiving a second medical image of        the examination volume, wherein the second medical image        corresponds to a second examination time being different from        the first examination time; and    -   a computation unit configured for determining a first lesion        area corresponding to the lesion within the first medical image;        furthermore configured for determining a registration function        based on a comparison of the first medical image and the second        medical image;        furthermore configured for determining a second lesion area        within the second medical image based on the registration        function and the first lesion area;        furthermore configured for classifying the lesion within the        first medical image based on the second lesion area.

According to a fourth embodiment, the invention relates to a medicalimaging system, comprising a classification system to the invention andits aspects. In particular, the medical imaging system can be an X-raybased medical imaging system, e.g. a computed tomography system or afluoroscopy system.

According to a fifth embodiment, the invention relates to a providingsystem comprising an interface and a calculation unit:

-   -   wherein the interface is configured for receiving a first        medical training image of a training examination volume, wherein        the first medical training image corresponds to a first        examination time,    -   wherein the interface is furthermore configured for receiving a        second medical training image of the training examination        volume, wherein the second medical training image corresponds to        a second examination time being different from the first        examination time,    -   wherein the calculation unit is configured for determining a        registration function based on a comparison of the first medical        training image and the second medical training image,    -   wherein the calculation unit is furthermore configured for        receiving a training classification corresponding to a first        lesion area within the first medical training image,    -   wherein the calculation unit is furthermore configured for        determining a second lesion area within the second medical        training image based on the registration function and the first        medical training image,    -   wherein the calculation unit is furthermore configured for        applying a trained classifying function to first training input        data and second training input data, thereby generating training        output data, wherein the first training input data is based on        the first lesion area, wherein the second training input data is        based on the second lesion area,    -   wherein the calculation unit is furthermore configured for        adjusting at least one parameter of the trained classifying        function based on a comparison of the training classification        and the output data,    -   wherein the interface is furthermore configured for providing        the trained classifying function (TCF).

According to a sixth embodiment, the invention relates to a computerprogram comprising instructions which, when the program is executed by aclassification system, cause the classification system to carry out themethod for classifying a lesion according to an embodiment of theinvention and its aspects.

According to a possible seventh embodiment, the invention relates to acomputer program comprising instructions which, when the program isexecuted by a providing system, cause the providing system to carry outthe method for providing a trained classification function according toan embodiment of the invention and its aspects.

According to an eighth embodiment, the invention relates to acomputer-readable medium comprising instructions which, when executed bya classification system, cause the classification system to carry outthe method for classifying a lesion according to an embodiment of theinvention and its aspects.

According to a possible ninth embodiment, the invention relates to acomputer-readable medium comprising instructions which, when executed bya providing system, cause the providing system to carry out the methodfor providing a trained classification function according to anembodiment of the invention and its aspects.

According to a tenth embodiment, the invention relates to acomputer-readable storage medium, comprising a trained classifyingfunction provided by a method for providing a trained classifyingfunction according to an embodiment of the invention and its aspects,for use in a method for classifying a lesion.

According to a possible eleventh embodiment, the invention relates to acomputer-readable storage medium, comprising a trained classifyingfunction provided by a method for providing a trained classifyingfunction according to an embodiment of the invention and its aspects.

According to a possible twelfth embodiment, the invention relates to amethod of using a computer-readable storage medium, comprising a trainedclassifying function provided by a method for providing a trainedclassifying function according to an embodiment of the invention and itsaspects, for classifying a lesion.

According to an embodiment, the invention relates to acomputer-implemented method for classifying a lesion, comprising:

receiving a first medical image of an examination volume, the firstmedical image corresponding to a first examination time;

receiving a second medical image of the examination volume, the secondmedical image corresponding to a second examination time, different fromthe first examination time;

determining a first lesion area corresponding to a lesion within thefirst medical image;

determining a registration function based on a comparison of the firstmedical image and the second medical image;

determining a second lesion area within the second medical image basedon the registration function and the first lesion area; and

classifying the lesion within the first medical image based on thesecond lesion area.

According to an embodiment, the invention relates to acomputer-implemented method for providing a trained classifyingfunction, comprising:

receiving a first medical training image of a training examinationvolume, the first medical training image corresponding to a firstexamination time;

receiving a second medical training image of the training examinationvolume, the second medical training image corresponding to a secondexamination time, different from the first examination time;

determining a first lesion area corresponding to a lesion within thefirst medical training image;

determining a registration function based on a comparison of the firstmedical training image and the second medical training image;

receiving a training classification corresponding to a first lesion areawithin the first medical training image;

determining a second lesion area within the second medical trainingimage, based on the registration function and the first medical trainingimage;

applying a trained classifying function to first training input data andsecond training input data, to generate training output data, the firsttraining input data being based on the first lesion area and the secondtraining input data being based on the second lesion area;

adjusting at least one parameter of the trained classifying functionbased on a comparison of the training classification and the trainingoutput data; and

providing the trained classifying function.

According to an embodiment, the invention relates to a classificationsystem for classifying a lesion, comprising:

an interface configured to

-   -   receive a first medical image of an examination volume, the        first medical image corresponding to a first examination time,    -   receive a second medical image of the examination volume, the        second medical image corresponding to a second examination time,        different from the first examination time; and

at least one processor configured to

-   -   determine a first lesion area corresponding to a lesion within        the first medical image,    -   determine a registration function based on a comparison of the        first medical image and the second medical image,    -   determine a second lesion area within the second medical image        based on the registration function and the first lesion area,        and    -   classify the lesion within the first medical image based on the        second lesion area.

According to an embodiment, the invention relates to a medical imagingsystem, comprising: the classification system of claim an embodiment.

According to an embodiment, the invention relates to a non-transitorycomputer program product storing instructions which, when the program isexecuted by a classification system, cause the classification system tocarry out the method of an embodiment.

According to an embodiment, the invention relates to a non-transitorycomputer-readable medium storing instructions which, when executed by aclassification system, cause the classification system to carry out themethod of an embodiment.

According to an embodiment, the invention relates to a non-transitorycomputer-readable storage medium, storing a trained classifying functionprovided by the method according to an embodiment, for use in a methodfor classifying a lesion upon execution by at least one processor.

BRIEF DESCRIPTION OF THE DRAWINGS

The properties, features and advantages of this invention describedabove, as well as the manner they are achieved, become clearer and moreunderstandable in the light of the following description andembodiments, which will be described in detail in the context of thedrawings. This following description does not limit the invention on thecontained embodiments. Same components or parts can be labeled with thesame reference signs in different figures. In general, the figures arenot for scale. In the following:

FIG. 1 displays a timeline indicating the relation between medicalimages and registration functions,

FIG. 2 displays a data flow diagram for embodiments of the method forclassifying a lesion,

FIG. 3 displays a recurrent neural network comprising LSTM blocks astrained classifying function,

FIG. 4 displays a first embodiment of the method for classifying alesion,

FIG. 5 displays a second embodiment of the method for classifying alesion,

FIG. 6 displays a third embodiment of the method for classifying alesion,

FIG. 7 displays a fourth embodiment of the method for classifying alesion,

FIG. 8 displays an embodiment of the method for providing a trainedclassifying function,

FIG. 9 displays a classification system,

FIG. 10 displays a training system.

DETAILED DESCRIPTION OF THE EXAMPLE EMBODIMENTS

The drawings are to be regarded as being schematic representations andelements illustrated in the drawings are not necessarily shown to scale.Rather, the various elements are represented such that their functionand general purpose become apparent to a person skilled in the art. Anyconnection or coupling between functional blocks, devices, components,or other physical or functional units shown in the drawings or describedherein may also be implemented by an indirect connection or coupling. Acoupling between components may also be established over a wirelessconnection. Functional blocks may be implemented in hardware, firmware,software, or a combination thereof.

Various example embodiments will now be described more fully withreference to the accompanying drawings in which only some exampleembodiments are shown. Specific structural and functional detailsdisclosed herein are merely representative for purposes of describingexample embodiments. Example embodiments, however, may be embodied invarious different forms, and should not be construed as being limited toonly the illustrated embodiments. Rather, the illustrated embodimentsare provided as examples so that this disclosure will be thorough andcomplete, and will fully convey the concepts of this disclosure to thoseskilled in the art. Accordingly, known processes, elements, andtechniques, may not be described with respect to some exampleembodiments. Unless otherwise noted, like reference characters denotelike elements throughout the attached drawings and written description,and thus descriptions will not be repeated. At least one embodiment ofthe present invention, however, may be embodied in many alternate formsand should not be construed as limited to only the example embodimentsset forth herein.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, components, regions,layers, and/or sections, these elements, components, regions, layers,and/or sections, should not be limited by these terms. These terms areonly used to distinguish one element from another. For example, a firstelement could be termed a second element, and, similarly, a secondelement could be termed a first element, without departing from thescope of example embodiments of the present invention. As used herein,the term “and/or,” includes any and all combinations of one or more ofthe associated listed items. The phrase “at least one of” has the samemeaning as “and/or”.

Spatially relative terms, such as “beneath,” “below,” “lower,” “under,”“above,” “upper,” and the like, may be used herein for ease ofdescription to describe one element or feature's relationship to anotherelement(s) or feature (s) as illustrated in the figures. It will beunderstood that the spatially relative terms are intended to encompassdifferent orientations of the device in use or operation in addition tothe orientation depicted in the figures. For example, if the device inthe figures is turned over, elements described as “below,” “beneath,” or“under,” other elements or features would then be oriented “above” theother elements or features. Thus, the example terms “below” and “under”may encompass both an orientation of above and below. The device may beotherwise oriented (rotated 90 degrees or at other orientations) and thespatially relative descriptors used herein interpreted accordingly. Inaddition, when an element is referred to as being “between” twoelements, the element may be the only element between the two elements,or one or more other intervening elements may be present.

Spatial and functional relationships between elements (for example,between modules) are described using various terms, including“connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitlydescribed as being “direct,” when a relationship between first andsecond elements is described in the above disclosure, that relationshipencompasses a direct relationship where no other intervening elementsare present between the first and second elements, and also an indirectrelationship where one or more intervening elements are present (eitherspatially or functionally) between the first and second elements. Incontrast, when an element is referred to as being “directly” connected,engaged, interfaced, or coupled to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between,” versus “directly between,” “adjacent,” versus“directly adjacent,” etc.).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of exampleembodiments of the invention. As used herein, the singular forms “a,”“an,” and “the,” are intended to include the plural forms as well,unless the context clearly indicates otherwise. As used herein, theterms “and/or” and “at least one of” include any and all combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “comprises,” “comprising,” “includes,” and/or“including,” when used herein, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items. Expressionssuch as “at least one of,” when preceding a list of elements, modify theentire list of elements and do not modify the individual elements of thelist. Also, the term “example” is intended to refer to an example orillustration.

When an element is referred to as being “on,” “connected to,” “coupledto,” or “adjacent to,” another element, the element may be directly on,connected to, coupled to, or adjacent to, the other element, or one ormore other intervening elements may be present. In contrast, when anelement is referred to as being “directly on,” “directly connected to,”“directly coupled to,” or “immediately adjacent to,” another elementthere are no intervening elements present.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which example embodiments belong. Itwill be further understood that terms, e.g., those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Before discussing example embodiments in more detail, it is noted thatsome example embodiments may be described with reference to acts andsymbolic representations of operations (e.g., in the form of flowcharts, flow diagrams, data flow diagrams, structure diagrams, blockdiagrams, etc.) that may be implemented in conjunction with units and/ordevices discussed in more detail below. Although discussed in aparticularly manner, a function or operation specified in a specificblock may be performed differently from the flow specified in aflowchart, flow diagram, etc. For example, functions or operationsillustrated as being performed serially in two consecutive blocks mayactually be performed simultaneously, or in some cases be performed inreverse order. Although the flowcharts describe the operations assequential processes, many of the operations may be performed inparallel, concurrently or simultaneously. In addition, the order ofoperations may be re-arranged. The processes may be terminated whentheir operations are completed, but may also have additional steps notincluded in the figure. The processes may correspond to methods,functions, procedures, subroutines, subprograms, etc.

Specific structural and functional details disclosed herein are merelyrepresentative for purposes of describing example embodiments of thepresent invention. This invention may, however, be embodied in manyalternate forms and should not be construed as limited to only theembodiments set forth herein.

Units and/or devices according to one or more example embodiments may beimplemented using hardware, software, and/or a combination thereof. Forexample, hardware devices may be implemented using processing circuitrysuch as, but not limited to, a processor, Central Processing Unit (CPU),a controller, an arithmetic logic unit (ALU), a digital signalprocessor, a microcomputer, a field programmable gate array (FPGA), aSystem-on-Chip (SoC), a programmable logic unit, a microprocessor, orany other device capable of responding to and executing instructions ina defined manner. Portions of the example embodiments and correspondingdetailed description may be presented in terms of software, oralgorithms and symbolic representations of operation on data bits withina computer memory. These descriptions and representations are the onesby which those of ordinary skill in the art effectively convey thesubstance of their work to others of ordinary skill in the art. Analgorithm, as the term is used here, and as it is used generally, isconceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of optical, electrical, or magnetic signals capable of beingstored, transferred, combined, compared, and otherwise manipulated. Ithas proven convenient at times, principally for reasons of common usage,to refer to these signals as bits, values, elements, symbols,characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, or as is apparent from the discussion,terms such as “processing” or “computing” or “calculating” or“determining” of “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computingdevice/hardware, that manipulates and transforms data represented asphysical, electronic quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

In this application, including the definitions below, the term ‘module’or the term ‘controller’ may be replaced with the term ‘circuit.’ Theterm ‘module’ may refer to, be part of, or include processor hardware(shared, dedicated, or group) that executes code and memory hardware(shared, dedicated, or group) that stores code executed by the processorhardware.

The module may include one or more interface circuits. In some examples,the interface circuits may include wired or wireless interfaces that areconnected to a local area network (LAN), the Internet, a wide areanetwork (WAN), or combinations thereof. The functionality of any givenmodule of the present disclosure may be distributed among multiplemodules that are connected via interface circuits. For example, multiplemodules may allow load balancing. In a further example, a server (alsoknown as remote, or cloud) module may accomplish some functionality onbehalf of a client module.

Software may include a computer program, program code, instructions, orsome combination thereof, for independently or collectively instructingor configuring a hardware device to operate as desired. The computerprogram and/or program code may include program or computer-readableinstructions, software components, software modules, data files, datastructures, and/or the like, capable of being implemented by one or morehardware devices, such as one or more of the hardware devices mentionedabove. Examples of program code include both machine code produced by acompiler and higher level program code that is executed using aninterpreter.

For example, when a hardware device is a computer processing device(e.g., a processor, Central Processing Unit (CPU), a controller, anarithmetic logic unit (ALU), a digital signal processor, amicrocomputer, a microprocessor, etc.), the computer processing devicemay be configured to carry out program code by performing arithmetical,logical, and input/output operations, according to the program code.Once the program code is loaded into a computer processing device, thecomputer processing device may be programmed to perform the programcode, thereby transforming the computer processing device into a specialpurpose computer processing device. In a more specific example, when theprogram code is loaded into a processor, the processor becomesprogrammed to perform the program code and operations correspondingthereto, thereby transforming the processor into a special purposeprocessor.

Software and/or data may be embodied permanently or temporarily in anytype of machine, component, physical or virtual equipment, or computerstorage medium or device, capable of providing instructions or data to,or being interpreted by, a hardware device. The software also may bedistributed over network coupled computer systems so that the softwareis stored and executed in a distributed fashion. In particular, forexample, software and data may be stored by one or more computerreadable recording mediums, including the tangible or non-transitorycomputer-readable storage media discussed herein.

Even further, any of the disclosed methods may be embodied in the formof a program or software. The program or software may be stored on anon-transitory computer readable medium and is adapted to perform anyone of the aforementioned methods when run on a computer device (adevice including a processor). Thus, the non-transitory, tangiblecomputer readable medium, is adapted to store information and is adaptedto interact with a data processing facility or computer device toexecute the program of any of the above mentioned embodiments and/or toperform the method of any of the above mentioned embodiments.

Example embodiments may be described with reference to acts and symbolicrepresentations of operations (e.g., in the form of flow charts, flowdiagrams, data flow diagrams, structure diagrams, block diagrams, etc.)that may be implemented in conjunction with units and/or devicesdiscussed in more detail below. Although discussed in a particularlymanner, a function or operation specified in a specific block may beperformed differently from the flow specified in a flowchart, flowdiagram, etc. For example, functions or operations illustrated as beingperformed serially in two consecutive blocks may actually be performedsimultaneously, or in some cases be performed in reverse order.

According to one or more example embodiments, computer processingdevices may be described as including various functional units thatperform various operations and/or functions to increase the clarity ofthe description. However, computer processing devices are not intendedto be limited to these functional units. For example, in one or moreexample embodiments, the various operations and/or functions of thefunctional units may be performed by other ones of the functional units.Further, the computer processing devices may perform the operationsand/or functions of the various functional units without subdividing theoperations and/or functions of the computer processing units into thesevarious functional units.

Units and/or devices according to one or more example embodiments mayalso include one or more storage devices. The one or more storagedevices may be tangible or non-transitory computer-readable storagemedia, such as random access memory (RAM), read only memory (ROM), apermanent mass storage device (such as a disk drive), solid state (e.g.,NAND flash) device, and/or any other like data storage mechanism capableof storing and recording data. The one or more storage devices may beconfigured to store computer programs, program code, instructions, orsome combination thereof, for one or more operating systems and/or forimplementing the example embodiments described herein. The computerprograms, program code, instructions, or some combination thereof, mayalso be loaded from a separate computer readable storage medium into theone or more storage devices and/or one or more computer processingdevices using a drive mechanism. Such separate computer readable storagemedium may include a Universal Serial Bus (USB) flash drive, a memorystick, a Blu-ray/DVD/CD-ROM drive, a memory card, and/or other likecomputer readable storage media. The computer programs, program code,instructions, or some combination thereof, may be loaded into the one ormore storage devices and/or the one or more computer processing devicesfrom a remote data storage device via a network interface, rather thanvia a local computer readable storage medium. Additionally, the computerprograms, program code, instructions, or some combination thereof, maybe loaded into the one or more storage devices and/or the one or moreprocessors from a remote computing system that is configured to transferand/or distribute the computer programs, program code, instructions, orsome combination thereof, over a network. The remote computing systemmay transfer and/or distribute the computer programs, program code,instructions, or some combination thereof, via a wired interface, an airinterface, and/or any other like medium.

The one or more hardware devices, the one or more storage devices,and/or the computer programs, program code, instructions, or somecombination thereof, may be specially designed and constructed for thepurposes of the example embodiments, or they may be known devices thatare altered and/or modified for the purposes of example embodiments.

A hardware device, such as a computer processing device, may run anoperating system (OS) and one or more software applications that run onthe OS. The computer processing device also may access, store,manipulate, process, and create data in response to execution of thesoftware. For simplicity, one or more example embodiments may beexemplified as a computer processing device or processor; however, oneskilled in the art will appreciate that a hardware device may includemultiple processing elements or processors and multiple types ofprocessing elements or processors. For example, a hardware device mayinclude multiple processors or a processor and a controller. Inaddition, other processing configurations are possible, such as parallelprocessors.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium (memory).The computer programs may also include or rely on stored data. Thecomputer programs may encompass a basic input/output system (BIOS) thatinteracts with hardware of the special purpose computer, device driversthat interact with particular devices of the special purpose computer,one or more operating systems, user applications, background services,background applications, etc. As such, the one or more processors may beconfigured to execute the processor executable instructions.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language) or XML (extensible markuplanguage), (ii) assembly code, (iii) object code generated from sourcecode by a compiler, (iv) source code for execution by an interpreter,(v) source code for compilation and execution by a just-in-timecompiler, etc. As examples only, source code may be written using syntaxfrom languages including C, C++, C#, Objective-C, Haskell, Go, SQL, R,Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5,Ada, ASP (active server pages), PHP, Scala, Eiffel, Smalltalk, Erlang,Ruby, Flash®, Visual Basic®, Lua, and Python®.

Further, at least one embodiment of the invention relates to thenon-transitory computer-readable storage medium including electronicallyreadable control information (processor executable instructions) storedthereon, configured in such that when the storage medium is used in acontroller of a device, at least one embodiment of the method may becarried out.

The computer readable medium or storage medium may be a built-in mediuminstalled inside a computer device main body or a removable mediumarranged so that it can be separated from the computer device main body.The term computer-readable medium, as used herein, does not encompasstransitory electrical or electromagnetic signals propagating through amedium (such as on a carrier wave); the term computer-readable medium istherefore considered tangible and non-transitory. Non-limiting examplesof the non-transitory computer-readable medium include, but are notlimited to, rewriteable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewriteable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. Shared processor hardware encompasses asingle microprocessor that executes some or all code from multiplemodules. Group processor hardware encompasses a microprocessor that, incombination with additional microprocessors, executes some or all codefrom one or more modules. References to multiple microprocessorsencompass multiple microprocessors on discrete dies, multiplemicroprocessors on a single die, multiple cores of a singlemicroprocessor, multiple threads of a single microprocessor, or acombination of the above.

Shared memory hardware encompasses a single memory device that storessome or all code from multiple modules. Group memory hardwareencompasses a memory device that, in combination with other memorydevices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium is therefore considered tangible and non-transitory. Non-limitingexamples of the non-transitory computer-readable medium include, but arenot limited to, rewriteable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewriteable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks andflowchart elements described above serve as software specifications,which can be translated into the computer programs by the routine workof a skilled technician or programmer.

Although described with reference to specific examples and drawings,modifications, additions and substitutions of example embodiments may bevariously made according to the description by those of ordinary skillin the art. For example, the described techniques may be performed in anorder different with that of the methods described, and/or componentssuch as the described system, architecture, devices, circuit, and thelike, may be connected or combined to be different from theabove-described methods, or results may be appropriately achieved byother components or equivalents.

According to a first embodiment, the invention relates to acomputer-implemented method for classifying a lesion, comprisingreceiving a first medical image of an examination volume, wherein thefirst medical image corresponds to a first examination time, furthermorecomprising receiving a second medical image of the examination volume,wherein the second medical image corresponds to a second examinationtime being different from the first examination time, furthermorecomprising determining a first lesion area corresponding to the lesionwithin the first medical image, furthermore comprising determining aregistration function based on a comparison of the first medical imageand the second medical image, determining a second lesion area withinthe second medical image based on the registration function and thefirst lesion area, and furthermore comprising classifying the lesionwithin the first medical image based on the second lesion area.

In particular, the steps of receiving the first medical image andreceiving the second medical image are executed by an interface, inparticular, by an interface of a classification system. In particular,the step of determining the first lesion area, of determining theregistration function, of determining the second lesion area and ofclassifying the lesion are executed by a computation unit, inparticular, by a computation unit of the classification system.

In particular, a lesion corresponds to a damage or a change in a tissueof an organism (in particular, a mammal, in particular, a human),usually caused by disease or trauma. In particular, a lesion can becaused by a tumor, and classified according to its benignancy and/or itsmalignancy.

A medical image corresponds to the result of a medical imagingexamination. A medical image can comprise additional data (inparticular, meta-data). In particular, a medical image can be an X-rayimage, a computed-tomography image, an ultrasound image, a magneticresonance image, a positron emission tomography image, a single-photonemission computed tomography image, and/or a digital pathology image. Inparticular, the first medical image and the second medical image are ofthe same type.

In particular, a medical image is two-dimensional medical image, athree-dimensional medical image and/or a four-dimensional medical image.A medical image can comprise a plurality of pixels or voxels (the terms“pixels” and “voxels” are used as synonyms within this specification andcorrespond to the elementary building blocks of an image). Inparticular, each pixel or voxel comprises an assigned intensity value.In particular, the intensity value of a pixel can correspond to an X-rayattenuation value of tissue mapped by the respective pixel or voxel.

In particular, each medical image corresponds to an examination time,being the point in time the medical image was created. The first medicalexamination time can be earlier in time than the second medicalexamination time, or vice versa.

In particular, a registration function is a function which maps an inputmedical image to an output medical image. In particular, theregistration function assigns for a subset of pixels or voxels of theinput medical image corresponding pixels or voxels in the output medicalimage. In particular, the input medical image and the output medicalimage have the same dimensionality. In particular, the first medicalimage is the input medical image, and the second medical image is theoutput medical image.

The registration function can be an intensity-based registrationfunction and/or a feature-based registration function. The registrationfunction can be based on a linear (or affine) transformation or based ona non-linear transformation. A non-linear transformation can be based onradial basis functions, physical continuum models and/or largedeformation models (e.g. diffeomorphisms). A registration function canbe based on a frequency-domain representation of the first and/or thesecond medical image, e.g. a Fourier or a Laplace transformation of thefirst and/or the second medical image. A registration function can bedetermined manually, interactively, semi-automatically or automatically.In particular, a registration function can be determined by applying atrained registration function (e.g. a convolutional or anon-convolutional neural network) based on known training registrationsof pairs of training images.

In particular, a lesion area is an area or a region in a medical imagecorresponding to a lesion. In particular, the lesion area can compriseall pixels or voxels of the medical image corresponding to the lesion.In particular, the first lesion area is a lesion area within the firstmedical image. In particular, the second lesion area is a lesion areawithin the second medical image. In particular, the first lesion areaand the second lesion area correspond to the same lesion. In particular,a lesion area can be a segmentation or a mask of a lesion within thecorresponding medical image.

The step of classifying a lesion can comprise determining aclassification value corresponding to a lesion (or the first and thesecond lesion area) and/or providing the classification value. Providingthe classification value can comprise storing, displaying ortransmitting the classification value.

The inventors recognized that by the classification being based on thesecond lesion area and the second medical image, longitudinalinformation of the disease pathway can be considered. By using aregistration function to determine the second lesion based on the firstlesion area, it is also possible to use the additional data even if fromthe second medical image alone a lesion area is not detectable (fully)by conventional methods.

According to a further possible embodiment of the invention, determiningthe second lesion area is based on applying the registration function tothe first lesion area. In particular, the pixels or voxels correspondingto the first lesion area are mapped to pixels or voxels within thesecond medical image, and the set of these pixels or voxels within thesecond medical image defines the second lesion area within the secondmedical image. In particular, the second lesion area defined by a set ofpixels and voxels can be the convex hull of said set of pixels orvoxels. Alternatively, the pixels or voxels corresponding to theboundary of the first lesion area are mapped to pixels or voxels withinthe second medical image, and the convex hull of the set of these pixelsor voxels within the second medical image defines the second lesion arewithin the second medical image.

The inventors recognized that by applying the registration function tothe first lesion area the second lesion area can be determined in a fastand effective manner. By relying on the convex hull, an efficient errorcorrection algorithm can be used for cases where some pixels or voxelsof the second medical image are not a mapping target due to numerical orrounding errors.

According to a further possible embodiment of the invention, the step ofclassifying the lesion is furthermore based on the first lesion area.The inventors recognized that by classifying the lesion being based onthe first lesion area and the second lesion area longitudinalinformation can be incorporated very efficiently into the classificationprocess.

According to a further embodiment of the invention, the step ofclassifying the lesion comprises the sub-step of applying a trainedclassifying function to first input data and second input data, therebygenerating output data, wherein the first input data is based on thefirst lesion area, wherein the second input data is based on the secondlesion area, and wherein the step of classifying the lesion comprisesthe sub-step of determining a lesion classification based on the outputdata. In particular, the lesion classification can be identical with theoutput data.

A trained classifying function is a trained function. In general, atrained function mimics cognitive functions that humans associate withother human minds. In particular, by training based on training data thetrained function is able to adapt to new circumstances and to detect andextrapolate patterns.

In general, parameters of a trained function can be adapted by means oftraining. In particular, supervised training, semi-supervised training,unsupervised training, reinforcement learning and/or active learning canbe used. Furthermore, representation learning (an alternative term is“feature learning”) can be used. In particular, the parameters of thetrained functions can be adapted iteratively by several steps oftraining.

In particular, a trained function can comprise a neural network, asupport vector machine, a decision tree and/or a Bayesian network,and/or the trained function can be based on k-means clustering,Q-learning, genetic algorithms and/or association rules. In particular,a neural network can be a deep neural network, a convolutional neuralnetwork or a convolutional deep neural network. Furthermore, a neuralnetwork can be an adversarial network, a deep adversarial network and/ora generative adversarial network.

The inventors recognized that using a trained classification functionall features corresponding to the first and the second lesion area canbe considered for creating the output data and the classification value.In particular, also correlations not recognized by a human expert can beutilized for inferring the output data and the classification value.

By the input of the trained classification function being based on thefirst lesion area and the second lesion area, correlations betweendifferent stages of the development of the lesion can be considered. Byconsidering such correlations, the lesion qualification can be betterand more exact than based only separately on medical images of differentpoints in time.

According to a further embodiment of the invention, wherein the trainedclassifying function is a recurrent neural network, wherein the firstinput data and the second input data are independently used as inputdata for the recurrent neural network.

In particular, a recurrent neural network is an artificial neuralnetwork where connections between nodes form a directed graph along atemporal sequence. In particular, a recurrent neural network can beinterpreted as directed acyclic graph. In particular, the recurrentneural network can be a finite impulse recurrent neural network or aninfinite impulse recurrent neural network (wherein a finite impulsenetwork can be unrolled and replaced with a strictly feedforward neuralnetwork, and an infinite impulse network cannot be unrolled and replacedwith a strictly feedforward neural network). In particular, therecurrent neural network can comprise additional storage states oradditional network structures that incorporate time delays or comprisefeedback loops.

Equivalently, a recurrent neural network could also be defined as aneural network whose output does not only depend on the input value andthe edge weights, but also on a hidden state vector, wherein the hiddenstate vector is based on previous inputs used on the recurrent neuralnetwork.

The inventors recognized that using a recurrent neural network variablelength sequences of inputs can be used. In particular, this implies thatthe method cannot be used only for a fixed number of medical images orlesion areas (and needs to be trained differently for every other numberof medical images or lesion areas used as input), but can be used for anarbitrary number of input medical images or input lesion areas. Thisimplies that the whole set of training data, independent of the numberof successive medical images or lesion areas, can be used within thetraining of the trained classification function, and that the trainingdata is not reduced to training data corresponding to a certain numberof successive medical images or lesion areas.

In particular, by using the first input data and the second input dataindependently as input to the recurrent neural network, information fromapplying the recurrent neural network to the second input data can bestored within the hidden state, and can be used as additionalinformation when applying the recurrent network to the first input data.This behavior leads to the output data and the classification valuebeing based on the sequence of lesion areas, so the classification ofthe lesion can be done with a high precision and taking into account thetime development of the lesion.

According to a further embodiment of the invention the recurrent neuralnetwork comprises at least one LSTM (acronym for “long short-termmemory”) block.

In particular, a LSTM block comprises a cell, an input gate, an outputgate and a forget gate, wherein the cell corresponds to the hiddenvector, and the input gate, the output gate and the forget gate regulatethe flow of information into and out of the cell. In particular, byusing a cell, LSTM blocks can prevent exploding and vanishing gradientproblems that can be encountered when training other types of recurrentneural networks.

The inventors recognized that LSTM blocks are suited for input data thatis separated by diverse or even unknown time intervals, and so aresuited for a use in longitudinal medical imaging studies, because thetime differences between different images in the longitudinal studiesare in general not constant, and may even be unknown.

According to a further embodiment of the invention, the step ofclassifying furthermore comprises the sub-step of determining the firstinput data by applying a trained preprocessing function to at least apart of the first medical image containing the first lesion area.Furthermore, the step of classifying comprises the sub-step ofdetermining the second input data by applying the trained preprocessingfunction to at least a part of the second medical image containing thesecond lesion area. In particular, the trained preprocessing function isa trained function.

The at least part of the first medical image containing the first lesionarea is a subset of the first medical image, wherein the subsetcomprises the first lesion area. The at least part of the second medicalimage containing the second lesion area is a subset of the secondmedical image, wherein the subset comprises the second lesion area.

In particular, the trained preprocessing function is applied to the atleast part of the first medical image by applying the trainedpreprocessing function to a pixel-wise or voxel-wise product of a maskdefined by the first lesion area and the first medical image. Inparticular, the trained preprocessing function is applied to at leastpart of the second medical image by applying the trained preprocessingfunction to a product of a mask defined by the second lesion area andthe second medical image.

The inventors recognized that by using trained preprocessing functionrelevant features of the first medical image and the second medicalimage, and/or the first lesion area and the second lesion area can beextracted from the respective objects before using the data as input tothe trained classification network. In particular, the trainedpreprocessing function can be used for normalization of the input data(e.g. to get input data with the same brightness and contrast).

According to a further embodiment of the invention, the trainedpreprocessing function is configured by training to classify lesionswithin single medical images. In particular, the trained preprocessingfunction is trained to classify lesions within single medical images byat least one of their parameters being adapted based on a comparison ofan training output and a real classification value of a lesion, whereinthe training output is the result of applying the trained preprocessingfunction to a training medical image. In particular, classifying basedon single medical images means that only medical images acquired at acertain time are used as input, and not longitudinal studies.

The inventors recognized that by using trained preprocessing functionsconfigured to classify lesions within single medical image known imageclassifications functions can be used as trained preprocessing functionin order to reduce the training effort for the trained classificationfunction.

According to a further possible embodiment of the invention, the trainedpreprocessing function is a convolutional neural network. In particular,the trained preprocessing function is a deep convolutional network. Theinventors recognized that a convolutional network and a deepconvolutional network are well-suited for image processing andextracting features from images.

According to a further embodiment of the invention, the registrationfunction is a non-rigid registration function. A synonym for “non-rigidregistration function” is “elastic registration function”.

In general, a rigid registration function preserves value of theEuclidean distance of two points, so that d(TF(x), TF(y)) d(x, y), whered(x, y) is the distance of two points, and TF(x) is the result ofapplying the registration function to a point. A non-rigid registrationfunction does not preserve the value of the Euclidean distance of twopoints.

In particular, the non-rigid registration function can be a radial basisfunction (in particular, selected from the group of thin-platetransformations or surface spline transformations, multiquadrictransformations, and compactly-supported transformations), physicalcontinuum models (e.g., viscous fluid models), and large deformationmodels (diffeomorphism transformations).

The inventors recognized that by using a non-rigid registration functionlocal geometric differences between the first medical image and thesecond medical image can be considered. In particular, those localgeometric differences can occur due to physical changes in theexamination volume in-between the first and the second medical imagingexamination, or due to different poses of the patient when performingthe first and the second medical imaging examination. In particular,non-rigid transformations can be helpful if a growing lesion effectschanges in the neighboring examination volume.

According to a further embodiment of the invention, determining theregistration function is based on a vector momentum-parameterizedstationary velocity field (an acronym is “vSVF”). Methods for vSVF areknown e.g. from Z. Shen et al., “Networks for Joint Affine andNon-Parametric Image Registration” (20019) 4219-4228.10.1109/CVPR.2019.00435, the entire contents of which are herebyincorporated herein by reference. In particular, vSVF is a fluid dynamicmethod that deforms the image according to a smooth velocity field,where the deformation map can be accumulated along the time.

The inventors recognized that using vSVF techniques registrationfunctions can be determined in a faster way, with a better control oftransformation regularity, than other comparable registrationtechniques.

According to a further embodiment of the invention, the first medicalimage and the second medical image are two-dimensional orthree-dimensional X-ray based medical images.

In particular, an X-ray based medical image is a medical image that wasrecorded by means of X-ray radiation. In particular, a three-dimensionalX-ray based medical image can be a computed tomography image.

The inventors recognized that the described methods are especially wellsuited for X-ray based medical images.

According to a second embodiment, the invention relates to acomputer-implemented method for providing a trained classifyingfunction. The method is based on receiving a first medical trainingimage of a training examination volume, wherein the first medicaltraining image corresponds to a first examination time, and on receivinga second medical training image of the training examination volume,wherein the second medical training image corresponds to a secondexamination time being different from the first examination time.Furthermore, the method is based on determining a registration functionbased on a comparison of the first medical training image and the secondmedical training image. Furthermore, the method is based on determininga first lesion area corresponding to a lesion within the first medicaltraining image. Furthermore, the method is based on receiving a trainingclassification corresponding to a first lesion area within the firstmedical training image. Furthermore, the method is based on determininga second lesion area within the second medical training image based onthe registration function and the first medical training image.Furthermore, the method is based on applying a trained classifyingfunction to first training input data and second training input data,thereby generating training output data, wherein the first traininginput data is based on the first lesion area, and wherein the secondtraining input data (TID.2) is based on the second lesion area (LA.2).Furthermore, the method is based on adjusting at least one parameter ofthe trained classifying function based on a comparison of the trainingclassification and the output data. Furthermore, the method is based onproviding the trained classifying function.

In particular, the steps of receiving the first medical training image,of receiving the second medical training image, of receiving thetraining classification and of providing the trained classifyingfunction are executed by an interface, in particular, by an interface ofa providing system. In particular, the steps of determining theregistration function, of determining the second lesion area, ofapplying the trained classifying function, and of adjusting at least oneparameter of the trained classifying function are executed by acomputation unit, in particular, by a computation unit of the providingsystem.

The inventors recognized that based on this method a trainedclassification function can be provided in an efficient way, wherein atthe same time the trained classification function can produce highquality results in classifying lesions in longitudinal studies.

According to a third embodiment, the invention relates to aclassification system comprising

-   -   an interface configured for receiving a first medical image of        an examination volume, wherein the first medical image        corresponds to a first examination time,        furthermore configured for receiving a second medical image of        the examination volume, wherein the second medical image        corresponds to a second examination time being different from        the first examination time; and    -   a computation unit configured for determining a first lesion        area corresponding to the lesion within the first medical image;        furthermore configured for determining a registration function        based on a comparison of the first medical image and the second        medical image;        furthermore configured for determining a second lesion area        within the second medical image based on the registration        function and the first lesion area;        furthermore configured for classifying the lesion within the        first medical image based on the second lesion area.

In particular, the classification system is configured for executing themethod for classifying a lesion according to the invention and itsaspects. In particular, the classification system is configured forexecuting the method for classifying a lesion by the interface and thecomputation unit being configured for executing the single steps of themethod for classifying a lesion.

According to a fourth embodiment, the invention relates to a medicalimaging system, comprising a classification system to the invention andits aspects. In particular, the medical imaging system can be an X-raybased medical imaging system, e.g. a computed tomography system or afluoroscopy system.

According to a fifth embodiment, the invention relates to a providingsystem comprising an interface and a calculation unit:

-   -   wherein the interface is configured for receiving a first        medical training image of a training examination volume, wherein        the first medical training image corresponds to a first        examination time,    -   wherein the interface is furthermore configured for receiving a        second medical training image of the training examination        volume, wherein the second medical training image corresponds to        a second examination time being different from the first        examination time,    -   wherein the calculation unit is configured for determining a        registration function based on a comparison of the first medical        training image and the second medical training image,    -   wherein the calculation unit is furthermore configured for        receiving a training classification corresponding to a first        lesion area within the first medical training image,    -   wherein the calculation unit is furthermore configured for        determining a second lesion area within the second medical        training image based on the registration function and the first        medical training image,    -   wherein the calculation unit is furthermore configured for        applying a trained classifying function to first training input        data and second training input data, thereby generating training        output data, wherein the first training input data is based on        the first lesion area, wherein the second training input data is        based on the second lesion area,    -   wherein the calculation unit is furthermore configured for        adjusting at least one parameter of the trained classifying        function based on a comparison of the training classification        and the output data,    -   wherein the interface is furthermore configured for providing        the trained classifying function (TCF).

According to a sixth embodiment, the invention relates to a computerprogram comprising instructions which, when the program is executed by aclassification system, cause the classification system to carry out themethod for classifying a lesion according to an embodiment of theinvention and its aspects.

According to a possible seventh embodiment, the invention relates to acomputer program comprising instructions which, when the program isexecuted by a providing system, cause the providing system to carry outthe method for providing a trained classification function according toan embodiment of the invention and its aspects.

According to an eighth embodiment, the invention relates to acomputer-readable medium comprising instructions which, when executed bya classification system, cause the classification system to carry outthe method for classifying a lesion according to an embodiment of theinvention and its aspects.

According to a possible ninth embodiment, the invention relates to acomputer-readable medium comprising instructions which, when executed bya providing system, cause the providing system to carry out the methodfor providing a trained classification function according to anembodiment of the invention and its aspects.

The realization of the invention by a computer program product and/or acomputer-readable medium has the advantage that already existingproviding systems can be easily adopted by software updates in order towork as proposed by the invention.

The computer program product can be, for example, a computer program orcomprise another element apart from the computer program. This otherelement can be hardware, for example a memory device, on which thecomputer program is stored, a hardware key for using the computerprogram and the like, and/or software, for example a documentation or asoftware key for using the computer program.

According to a tenth embodiment, the invention relates to acomputer-readable storage medium, comprising a trained classifyingfunction provided by a method for providing a trained classifyingfunction according to an embodiment of the invention and its aspects,for use in a method for classifying a lesion.

According to a possible eleventh embodiment, the invention relates to acomputer-readable storage medium, comprising a trained classifyingfunction provided by a method for providing a trained classifyingfunction according to an embodiment of the invention and its aspects.

According to a possible twelfth embodiment, the invention relates to amethod of using a computer-readable storage medium, comprising a trainedclassifying function provided by a method for providing a trainedclassifying function according to an embodiment of the invention and itsaspects, for classifying a lesion.

FIG. 1 displays the relation between medical images IMG.1, IMG.2, IMG.3and registration functions TF(t.2, t.1), TF(t.3, t.1).

The first medical image IMG.1 corresponds to a first examination timet.1, the second medical image IMG.2 corresponds to a second examinationtime t.2, and the third medical image IMG.3 corresponds to a thirdexamination time t.3. The first medical examination time t.1 is laterthan the second medical examination time t.2, and the second medicalexamination time t.2 is later than the third medical examination timet.3.

Each registration function TF(t.2, t.1), TF(t.3, t.1) is based on a pairof medical images IMG.1, IMG.2, IMG.3. In particular, each registrationfunction TF(t.2, t.1), TF(t.3, t.1) is based on a pair of medical imagesIMG.1, IMG.2, IMG.3 comprising the first medical image IMG.1 and anothermedical image IMG.2, IMG.3. In particular, the registration functionmaps coordinates of the first medical image IMG.1 to coordinates of theanother medical image IMG.2, IMG.3. The coordinates can be representedby real numbers, and/or by integer numbers corresponding to pixels. Inparticular, the registration function maps pixels or voxels of the firstmedical image to pixels or voxels of the second medical image.

For example, if the first medical image IMG.1 and the second medicalimage IMG.2 are two-dimensional medical images, the registrationfunction TF(t.2, t.1) maps a pixel P.1 of the first medical image IMG.1coordinatized by coordinates P.1=(i.1, j.1) to a pixel P.2 of the secondmedical image IMG.2 coordinatized by coordinates P.2=(i.2, j.2) by (i.2,j.2)=TF(t.2, t.1) (i.1, j.1).

If the first medical image IMG.3 and the second medical image IMG.3 arethree-dimensional medical images, the registration function TF(t.2, t.1)maps a voxel P.1 of the first medical image IMG.1 coordinatized bycoordinates P.1=(i.1, j.1, k.1) to a voxel P.2 of the second medicalimage IMG.2 coordinatized by coordinates P.2 (i.2, j.2, k.2) by (i.2,j.2, k.2)=TF(t.2, t.1) (i.1, j.1, k.1).

The registration function TF(t.2, t.1) can also be used for mapping afirst area within the first medical image IMG.1 to a second area withinthe second medical image IMG.2, wherein an area is e.g. by a set ofpixels or voxels of the respective image. For example, the first areacomprising pixels of first medical image IMG.1 is mapped to a set ofpixels in the second medial image IMG.2 forming the second area in thesecond medical image IMG.2, wherein the second area in the secondmedical image comprises the resulting pixels of applying theregistration function TF(t.2, t.1) pixel-wise to the pixels of the firstarea.

FIG. 2 displays a data flow diagram for some embodiments of the methodfor classifying a lesion. In this data flow diagram, three medicalimages IMG.1, IMG.2, IMG.3 are used as input, wherein a registrationfunction TF(t.2, t.1) for the first medical image IMG.1 and the secondmedical image IMG.2 and a registration function TF(t.3, t.1) for thefirst medical image IMG.1 and the third medical image IMG.3 iscalculated as displayed in FIG. X.

The first medical image IMG.1 comprises a first lesion area LA.1,comprising several pixels or voxels of the first medical image IMG.1.The registration functions TF(t.2, t.1), TF(t.3, t.1) are used todetermine a second lesion area LA.2 within the second medical imageIMG.2 and a third lesion area LA.3 within the third medical image IMG.3,in particular by LA.2=TF (t.2, t.1) (LA.1) und LA.3=TF (t.3, t.1)(LA.1).

In this embodiment, the lesion areas LA.1, LA.2, LA.3 are used as inputsfor a trained preprocessing function TPF in order to determine inputdata ID.1, ID.2, ID.3 by ID.i=TPF(LA.i). In particular, the trainedpreprocessing function TPF is trained convolutional neural network. Thelesion areas LA.1, LA.2, LA.3 can be used as inputs for the trainedpreprocessing function TPF also in combination with the respectivemedical image IMG.1, IMG.2, IMG.3, e.g. by using a pixel-wise orvoxel-wise product of the lesion area LA.1, LA.2, LA.3 being a maskimage and the respective medical image IMG.1, IMG.2, IMG.3 byID.i=(LA.i×IMG.i), wherein × denotes the pixel-wise or the voxel-wiseproduct. Alternatively, no trained preprocessing function TPF needs tobe used, in this case the input data ID.1, ID.2, ID.3 is equivalent withthe lesion area LA.1, LA.2, LA.3 (or, e.g., with the pixel-wise orvoxel-wise product of the lesion area LA.1, LA.2, LA.3 as mask and therespective medical image IMG.1, IMG.2, IMG.3).

The input data ID.1, ID.2, ID.3 is then used as input for a trainedclassification function TCF. In this embodiment, the trainedclassification function TCF is a recurrent neural network comprisingrecurrent neural network blocks RNB.1, RNB.2, RNB.3. In particular, eachinput data ID.1, ID.2, ID.3 is used as input for a different one of therecurrent neural network blocks RNB.1, RNB.2, RNB.3.

Every recurrent neural network block RNB.1, RNB.2, RNB.3 determinesoutput data OD.1, OD.2, OD.3 and intermediate data BD.1, BD.2, BD.3,wherein the intermediate data BD.1, BD.2, BD.3 is used as additionalinput for the next of the successive recurrent neural network blocksRNB.1, RNB.2, RNB.3.

The classification value of the lesion can then be based on all of theoutput data OD.1, OD.2, OD.3, or only on some of the output data OD.1,OD.2, OD.3. In particular, the classification value of the lesion isbased on the output data OD.1 related to the first medical image IMG.1,and potentially on further output data OD.2, OD.3. By being based atleast on the output data OD.1 corresponding to the first medical imageIMG.1, the most recent one of the medical images IMG.1, IMG.2, IMG.3 canbe used with a high weight in the classification of the lesion, implyinga better and more exact classification.

FIG. 3 displays a detailed view of an LSTM network comprising severalrecurrent neural network blocks RNB.i, RNB.j. Each recurrent neuralnetwork block RNB.i, RNB.j uses input data ID.i, ID.j to generate orcalculate output data OD.i, OD.j. Additionally, each recurrent neuralnetwork block RNB.i, RNB.j takes as additional input intermediate dataIBD.i, IBD.j and produces as additional output intermediate data OBD.i,OBD.j, wherein output intermediate data OBD.i, OBD.j can be used asinput intermediate data IBD.i, IBD.j within the next step.

It is important to understand that FIG. 3 displays an iterative process,which was unfolded for two inputs. In order to adapt for more inputdata, the iteration can be extended to cover an arbitrary number ofinput data ID.i, ID.j. Furthermore, the recurrent neural network blocksRNB.i, RNB.j are the same up to a number of internal states IG.i, IG.j,OG.i, OG.j, FG.i, FG.j. In particular, this implies that the output of aneural network block RNB.i, RNB.j only depends on the input data ID.i,ID.j, the additional input intermediate data IBD.i, IBD.j and theinternal states IG.i, IG.j, OG.i, OG.j, FG.i, FG.j.

In this embodiment, the neural network is an LSTM network, and therecurrent neural network block RNB.i, RNB.j has internal states denotedas input gate IG.i, IG.j, output gate OG.i, OG.j and forget gate FG.i,FG.j. In particular, the value of these internal states can becalculated asi _(j)=σ(W ^((x,I)) *x _(j) +W ^((y,I)) *y _(i) +W ^((c,I)) ·c _(i) +b^((I)))f _(j)=σ(W ^((x,F)) *x _(j) +W ^((y,F)) *y _(i) +W ^((c,F)) ·c _(i) +b^((F)))o _(j)=σ(W ^((x,O)) *x _(j) +W ^((y,O)) *y _(i) +W ^((c,O)) ·c _(j) +b^((O)))c _(j) =f _(j) ·c _(i) +i _(j)·tanh(W ^((x,C)) *x _(j) +W ^((y,C)) *y_(i) +b ^((C)))y _(j) =o _(j)·tanh(c _(j))

Within this iteration, the operation “·” is a pointwise multiplication,“*” is a convolution operation, and “σ” denotes the Sigmoid function.The values i_(j), o_(j) and f_(j) correspond to the values of the inputgate IG.j, the output gate OG.j and the forget gate FG.j. The valuesx_(j) and y_(j) correspond to the input data ID.j and the output dataOD.j of the respective block. The values c_(i) and c_(j) correspond tothe intermediate input intermediate data IBD.i and the outputintermediate data OBD.i, OBD.j, and are often denoted as “cell state”.The values W and b correspond to weights of the network, which are fixedby training the recurrent neural network.

In an alternative embodiment, one can simplify the update by not lettingthe cell state influence the updating of the input gate IG.i, IG.j, theoutput gate OG.i, OG.j and the forget gate FG.i, FG.j:i _(j)=σ(W ^((x,I)) *x _(j) +W ^((y,I)) *y _(i) +b ^((I)))f _(j)=σ(W ^((x,F)) *x _(j) +W ^((y,F)) *y _(i) +b ^((F)))o _(j)=σ(W ^((x,O)) *x _(j) +W ^((y,O)) *y _(i) +b ^((O)))c _(j) =f _(j) ·c _(i) +i _(j)·tanh(W ^((x,C)) *x _(j) +W ^((y,C)) *y_(i) +b ^((C)))y _(j) =o _(j)·tanh(c _(j))

In another alternative embodiment, the calculation of the cell state canbe modified in the following way:c _(j) =f _(j) ·c _(i)+(1−f _(j))·tanh(W ^((x,C)) *x _(j) +W ^((y,C)) *y_(i) +b ^((C)))

FIG. 4 displays a first embodiment of the computer-implemented methodfor classifying a lesion.

The first two steps of the displayed embodiment are receiving REC-IMG.1a first medical image IMG.1 of an examination volume and receivingREC-IMG.2 a second medical image IMG.2 of the examination volume. Thefirst medical image IMG.1 corresponds to a first examination time t.1(which is the time the first medical image IMG.1 was determined based ona medical imaging examination), and the second medical image IMG.2corresponds to a second examination time t.2 (which is the time thesecond medical image IMG.2 was determined based on another medicalimaging examination). Here, the second examination time t.2 is earlierthan the first examination time t.1, and the second examination time andthe first examination time are different. In particular, the first andthe second medical image IMG.1, IMG.2 can be part of a longitudinalstudy relating to a certain patient.

In this embodiment, both the first and the second medical image IMG.1,IMG.2 are three-dimensional images generated by means of a computedtomography imaging examination. Alternatively, the first and/or thesecond medical image IMG.1, IMG.2 can be based on other known method ofmedical imaging, e.g. magnetic resonance imaging, positron emissiontomography or single photon emission computed tomography. In particular,both the first and the second medical image IMG.1, IMG.2 are DICOMimages (acronym for “Digital Imaging and Communications in Medicine”).In particular, both the first and the second medical image IMG.1, IMG.2have the same size (measured in terms of number of voxels) with respectto each of the dimensions of the respective medical images IMG.1, IMG.2.

A next step of the displayed first embodiment is determining DET-LA.1 afirst lesion area LA.1 corresponding to a lesion within the firstmedical image IMG.1. In this embodiment, the lesion area corresponds toa segmentation of the lesion within the first medical image IMG.1. Inparticular, all voxels that are part of a segment corresponding to thelesion are considered as being part of the first lesion area LA.1, andall other voxels are considered as being not part of the first lesionarea LA.1.

There are various possibilities for determining DETLA.1 a first lesionarea LA.1 within a first medical image IMG.1 known to the person skilledin the art. In this embodiment, the method described in S. Ren et al.“Faster RCNN: Towards real-time object detection with region proposalnetworks” Advances in Neural Information Processing Systems 28 (NIPS2015) is used), the entire contents of which are hereby incorporatedherein by reference.

A further step of the displayed embodiment is determining DET-RF aregistration function RF(t.2, t.1) based on a comparison of the firstmedical image IMG.1 and the second medical image IMG.2. This furtherstep can be executed before, after or in parallel with the step ofdetermining DET-LA.1 the first lesion area LA.1.

In this embodiment, the registration function RF(t.2, t.1) is anon-rigid registration function based on a vector momentum-parameterizedstationary velocity field. Methods for determining such a registrationfunction RF(t.2, t.1) are known e.g. from the paper Z. Shen et al.“Networks for Joint Affine and Non-Parametric Image Registration”(20019) 4219-4228, 10.1109/CVPR.2019.00435.

In particular, if v denotes the vector field, Φ⁻¹ denotes theregistration function RF(t.2, t.1) (also denoted as registration map),I₁ denotes the first medical image IMG.1 and I₂ denotes the secondmedical image IMG.2, the registration function RF(t.2, t.1) can bedetermined by minimizingm*=argmin_(m0)λ_(vf) <m ₀ ,v ₀>+sim[I ₁∘Φ⁻¹(1),I ₂].The initial or boundary conditions are given by:Φ⁻¹ _(t) +DΦ ⁻¹ v=0;Φ⁻¹(0)=Φ⁻¹ ₍₀₎ ;v ₀=(L ⁺ L)⁻¹ m ₀.

Here, D denotes the Jacobian and v0=<L⁺Lv, v> is a spatial norm definedby specifying the differential operator L and its adjoint L⁺. Picking aspecific L implies picking an expected model of deformation. In vSVF,the differential operator is spatially invariant and is predefined toencode a desired level of smoothness. The vector-valued momentum m isboth spatio-temporal invariant and is equivalent to m=L⁺Lv. Theoptimization takes places on m, where the velocity is smoothed from it.The resulting transformation is guaranteed to be diffeomorphic.

The next step of the displayed first embodiment is determining DET-LA.2a second lesion area LA.2 within the second medical image IMG.2 based onthe registration function RF(t.2, t.1) and the first lesion area LA.1.Within this first embodiment, the registration function RF(t.2, t.1) isused to map every voxel of the first medical image IMG.1 correspondingto the first lesion area LA.1 within the first medical image IMG.1 to atleast one voxel of the second medical image IMG.2. All the voxels of thesecond medical image IMG.2 which are the result of such mapping are atleast a part of the second lesion area LA.2 within the second medicalimage IMG.2. Further normalization procedures can be applied, e.g. thesecond lesion area LA.2 can be defined as the convex hull of all thevoxels of the second medical image IMG.2 which are the result of suchmapping.

The last step of the displayed second embodiment is classifying CLF thelesion within the first medical image IMG.1 based on the second lesionarea LA.2. Within the second embodiment, the step of classifying CLF thelesion is furthermore based on the first lesion area LA.1. Within thisembodiment, the first lesion area LA.1 and the second lesion area LA.2(or, equivalently, the first and the second medical image IMG.1, IMG.2with the first and the second lesion area LA.1, LA.2 being used asmasks) are used as an input of a trained classification function TCF,which creates as an output a classification value of the lesion (e.g. abinary value whether the lesion is benign or malign).

FIG. 5 displays a second embodiment of the computer-implemented methodfor classifying a lesion. The steps of receiving REC-IMG.1, REC-IMG.2the first and the second medical image IMG.1, IMG.2, the step ofdetermining DET-RF the registration function RF(t.2, t.1), and the stepsof determining DET-LA.1, DET-LA.2 the first and the second lesion areaLA.1, LA.2 are equivalent to the corresponding steps of the firstembodiment and can comprise all advantageous embodiments of therespective steps.

In the second embodiment, the step of classifying CLF the lesioncomprises the substep of applying APL-TCF a trained classifying functionTCF to first input data ID.1 and second input data ID.2, therebygenerating output data OD.1, OD.2. Herein the first input data ID.1 isbased on the first lesion area LA.1, and the second input data ID.2 isbased on the second lesion area LA.2. In particular, the first inputdata ID.1 is based on a mask defined by the first lesion area LA.1 andthe first medical image IMG.1, and the second input data ID.2 is basedon a mask defined by the second lesion areas LA.2 and the second medicalimage IMG.2.

In this embodiment, the trained classifying function TCF is a recurrentneural network. The recurrent network can be applied to a sequence ofinput data ID.1, ID.2 by iteratively applying a block (comprising a ordefined by a neural network) to an each input data ID.1, ID.2, keepingan internal state. For example, if B denotes such a block, x_(n) denotesthe n-th input data ID.1, ID.2, then (y_(n), h_(n))=B(x_(n), h_(n+1)),wherein y_(n) denotes the n-th output data OD.1, OD.2 and h_(n) denotesthe hidden state vector after the n-th application of the block B of therecurrent neural network. The notation B_(HV)(x_(n), h_(n+1))=h_(n) canrefer to the hidden vector part of the output of the block B. Theinitial value h of the hidden state vector can be initialized randomlyor by a certain sequence (e.g., all entries can be set to 0 or 1).

Note that the iteration is defined “backwards”, since the first inputdata ID.1 (denotes as x1) corresponds to the last data in the temporalsequence of the longitudinal study.

Specifically, if there are a first and a second medical image IMG.1,IMG2, the second output data OD.2 can be calculated by (y₂, h₂)=B(x₂, h)and the first output data OD.1 can be calculated by (y₁, h₁)=B(x₁,h₂)=B(x₁, B_(HV)(x₂, h))

The step of classifying the lesion furthermore comprises the substep ofdetermining DET-LC a lesion classification based on the output dataOD.1, OD.2. In this embodiment, the lesion classification is a realnumber between 0 and 1, corresponding to the probability of theclassified lesion being malign. Alternatively, the lesion classificationcan be a class label indicating a type of a lesion. In particular, thelesion classification can be equivalent to the output data OD.1corresponding to the first medical image IMG.1.

FIG. 6 displays a third embodiment of the computer-implemented methodfor classifying a lesion. The steps of receiving REC-IMG.1, REC-IMG.2the first and the second medical image IMG.1, IMG.2, the step ofdetermining DET-RF the registration function RF(t.2, t.1), the steps ofdetermining DET-LA.1, DET-LA.2, the first and the second lesion areaLA.1, LA.2, as well as the substeps of applying APL-TCF a trainedclassifying function TCF to first input data ID.1 and second input dataID.2 and of determining DET-LC a lesion classification are equivalent tothe corresponding steps of the first embodiment and/or of the secondembodiment and can comprise all advantageous embodiments of therespective steps.

Within the third embodiment, the step of classifying CLF the lesioncomprises the substep of determining DET-ID.1 the first input data ID.1by applying a trained preprocessing function TPF to at least a part ofthe first medical image IMG.1 containing the first lesion area LA.1, andthe substep of determining DET-ID.2 the second input data ID.2 byapplying the trained preprocessing function TPF to at least a part ofthe second medical image IMG.2 containing the second lesion area LA.2.

In this embodiment, the trained preprocessing function TPF is aconvolutional deep neural network. In particular, the trainedpreprocessing function TPF is applied to a pixel-wise or voxel-wisemultiplication of the respective lesion area LA.1, LA.2 beinginterpreted as mask and the respective medical image IMG.1, IMG.2.

If I_(n) denotes the n-th medical image IMG.1, IMG.2, and L_(n)corresponds to the n-th lesion area LA.1, LA.2, wherein L_(n) has thesame dimensionality and the same size as I_(n), and L_(n)=0 for a pixelor voxel not being part of the respective lesion area, and L_(n)=1 for apixel or voxel being part of the respective lesion area, then the n-thinput data ID.1, ID.2 can be calculated as x_(n)=TPF(I_(n)·L_(n)).

In particular, within the notation of the second embodiment, if thereare a first and a second medical image IMG.1, IMG2, the second outputdata OD.2 can be calculated by (y₂, h₂)=B(TPF(I₂·L₂), h) and the firstoutput data OD.1 can be calculated by (y₁, h₁)=B(TPF(I₁·L₁),h2)=B(TPF(I₁·L₁), B_(HV)(TPF(I₂·L₂), h)).

FIG. 7 displays a fourth embodiment of the computer-implemented methodfor classifying a lesion. The steps of receiving REC-IMG.1, REC-IMG.2the first and the second medical image IMG.1, IMG.2, the step ofdetermining DET-LA.1 the first lesion area LA.1, the step of determiningDET-LA.2 the second lesion area LA.2 and the step of classifying thelesion can comprise all alternatives and advantageous embodiments of thefirst, the second and the third embodiment of the method for classifyinga lesion above.

The fourth embodiment displays the extension of the method for threemedical images IMG.1, IMG.2, IMG.3 within a longitudinal study. Thethird medical image IMG.3 is of the same type as the first and thesecond medical image IMG.1, IMG.2. The third medical image IMG.3corresponds to a third time t.3, wherein the third time t.3 is earlierthan the second time.

In contrast to the previous embodiments, the fourth embodiment alsocomprises the step of receiving REC-IMG.3 the third medical image IMG.3.If there would be additional medical images, there would be anadditional step of receiving for each of the additional medical images.

Furthermore, in contrast to the previous embodiments, within the fourthembodiment the step of determining DET-RF a registration functionRF(t.2, t.1), RF(t.3, t.1) comprises for each of the medical imagesIMG.2, IMG.3 except the first medical image IMG.1 a substep ofdetermining DET-RF.2, DETRF.3 a registration function RF(t.2, t.1),RF(t.3, t.1) based on the first medical image IMG.1 and the respectiveother medical image IMG.2, IMG.3. In this fourth embodiment, the step ofdetermining DET-RF a registration function RF(t.2, t.1), RF(t.3, t.1)comprises a substep of determining a first registration function RF(t.2,t.1) based on the first medical image IMG.1 and the second medical imageIMG.2 and a substep of determining a second registration functionRF(t.3, t.1) based on the first medical image IMG.1 and the thirdmedical image IMG.3. If there would be additional medical images, therewould be an additional substep of determining a registration functionbased on the first medical image and the respective additional medicalimage.

The fourth embodiment also comprises an additional step of determiningDET-LA.3 a third lesion area LA.3 within the third medical image IMG.3.The step of determining DETLA.3 the third lesion area LA.3 is equivalentto the step of determining DET-LA.2 the second lesion area LA.2, whereinthe second medical image IMG.2 is replaced with the third medical imageIMG.3 (and the corresponding registration function RF(t.3, t.1) isused). If there would be additional medical images, for each of theadditional medical images there would be an additional step ofdetermining a lesion area.

Furthermore, the step of classifying CLF is based on all the lesionareas LA.2, LA.3.

FIG. 8 displays a flow-chart of an embodiment of a computer-implementedmethod for providing a trained classifying function TCF.

The first steps of the displayed embodiments are receiving TREC-TIMG.1 afirst medical training image of a training examination volume, whereinthe first medical training image corresponds to a first examinationtime, and receiving a second medical training image of the trainingexamination volume, wherein the second medical training imagecorresponds to a second examination time being different from the firstexamination time. The first and the second medical training image cancomprise all advantageous features described in context of the first andthe second medical training image with respect to FIG. 4 .

In particular, both the first medical training image and the secondmedical training image are two-dimensional or three-dimensional X-raybased medical images.

Further steps of the displayed embodiment are determining TDET-LA.1 afirst lesion area corresponding to a lesion within the first medicaltraining image, determining TDET-RF a registration function based on acomparison of the first medical training image and the second medicaltraining image, and determining a second lesion area within the secondmedical training image based on the registration function and the firstmedical training image. These three steps can comprise all advantageousfeatures and embodiments described with the respective steps for theembodiments of the method for classifying a lesion displayed in one ofFIG. 4 to FIG. 7 .

Advantageously, the registration function is a non-rigid registrationfunction. In particular, determining TDET-RF the registration functionis based on a vector momentum-parameterized stationary velocity field.In particular, the registration function can be determined as describedwith respect to FIG. 4 and the method for classifying a lesion.

A further step of the displayed embodiment is receiving TREC-TCL atraining classification corresponding to the first lesion area withinthe first medical training image. In particular, the trainingclassification can be a binary value corresponding to whether the lesioncorresponding to the first lesion area is a malign lesion or a benignlesion. Alternatively, the training classification can be a class labelcorresponding to the class of the lesion corresponding to the firstlesion area, e.g. a BI-RADS (acronym for “Breast Imaging Reporting andData System”) classification value.

A further step of the displayed embodiment is applying TAPL-TCF atrained classifying function TCF to first training input data and secondtraining input data, thereby generating training output data, whereinthe first training input data is based on the first lesion area, andwherein the second training input data is based on the second lesionarea.

The relation between the first training input data, the lesion area andthe first medical training image is equivalent to the relation betweenthe first input data, the first lesion area and the first medical imagedescribed with respect to the method for classifying a lesion. Therelation between the second training input data, the second lesion areaand the second medical training image is equivalent to the relationbetween the second input data, the second lesion area and the secondmedical image described with respect to the method for classifying alesion.

Furthermore, the trained classifying function can comprise alladvantageous features and embodiments as the trained classifyingfunction described with respect to the method of classifying a lesion.

Advantageously, the trained classifying function TCF is a recurrentneural network, and the first training input data and the secondtraining input data are independently used as training input data forthe recurrent neural network. Advantageously, the recurrent neuralnetwork comprises at least one LSTM block.

Advantageously, the step of applying TAPL-TCF the trained classifyingfunction TCF furthermore comprises the substeps of determining the firsttraining input data by applying a trained preprocessing function TPF toat least a part of the first medical training image containing the firstlesion area, and determining the second training input data by applyingthe trained preprocessing function TPF to at least a part of the secondmedical training image containing the second lesion area. In particular,the trained preprocessing function is configured by training to classifylesions within single medical images, and/or wherein the trainedpreprocessing function TPF is a convolutional neural network.

In particular, the trained preprocessing function TPF can be pretrainedbased on a comparison between training classifications of lesions withina single medical image, and the result of its application to thesesingle medical images.

A further step of the displayed embodiment of the method for providing atrained classifying function is adjusting TADJ-TCF at least oneparameter of the trained classifying function TCF based on a comparisonof the training classification and the training output data. Inparticular, the parameter is adapted by minimizing a cost function,wherein the training classification and the trained output parameter areused as input values for the cost function. In particular, the costfunction has a (local or global) extremum if the training classificationequals the training output data. In particular, adjusting TADJ-TCF atleast one parameter of the trained classifying function TCF can comprisea gradient descent algorithm and/or a backpropagation algorithm.

The last step of the displayed embodiment is providing TPROV-TCF thetrained classifying function TCF. In particular, providing TPROV-TCF thetrained classifying function TCF can comprise displaying, storing and/ortransmitting the trained classifying function TCF.

FIG. 9 displays a classification system CSYS, FIG. 10 displays atraining system TSYS. The classification system CSYS comprises aninterface CSYS.IF, a computation unit CSYS.CU, and a memory unitCSYS.MU. The training system TSYS comprises an interface TSYS.IF, acomputation unit TSYS.CU, and a memory unit TSYS.MU.

The classification system CSYS and/or the training system TSYS can be a(personal) computer, a workstation, a virtual machine running on hosthardware, a microcontroller, or an integrated circuit. In particular,the classification system CSYS and/or the training system TSYS can bemobile devices, e.g. a smartphone or a tablet. As an alternative, theclassification system CSYS and/or the training system TSYS can be a realor a virtual group of computers (the technical term for a real group ofcomputers is “cluster”, the technical term for a virtual group ofcomputers is “cloud”).

An interface CSYS.IF, TSYS.IF can be embodied as a hardware interface oras a software interface (e.g. PCIBus, USB or Firewire). In general, acomputation unit CU.1, CU.2, CU.3 can comprise hardware elements andsoftware elements, for example a microprocessor, a CPU (acronym for“central processing unit”), a GPU (acronym for “graphical processingunit”), a field programmable gate array (an acronym is “FPGA”) or anASIC. (acronym for “application-specific integrated circuit”). Thecomputation unit CSYS.CU, TSYS.CU can be configured for multithreading,i.e. the calculation unit CSYS.CU, TSYS.CU can host differentcalculation processes at the same time, executing the either in parallelor switching between active and passive calculation processes. A memoryunit CSYS.MU, TSYS.MU can be e.g. nonpermanent main memory (e.g. randomaccess memory) or permanent mass storage (e.g. hard disk, USB stick, SDcard, solid state disk).

An interface CSYS.IF, TSYS.IF can comprise several (potentiallyspatially) separate sub-interfaces, each having the characteristics ofan interface described above. A computation unit CSYS.CU, TSYS.CU cancomprise several (potentially spatially) separate computation sub-units,each having the characteristics of a computation unit described above. Amemory unit CSYS.MU, TSYS.MU can comprise several (potentiallyspatially) separate memory sub-units, each having the characteristics ofa memory unit described above.

The classification system CSYS is configured to executed one of theembodiments of the method for classifying a lesion. In particular, theclassification CSYS is configured to execute this method by theinterface CSYS.IF and the computation unit CSYS.CU being configured toexecute the respective steps of the method. The training system TSYS isconfigured to execute one of the embodiments of the method for providinga trained classifying function TCF. In particular, the training systemTSYS is configured to execute this method by the interface TSYS.IF andthe computation unit TSYS.CU being configured to execute the respectivesteps of the method.

Wherever not already described explicitly, individual embodiments, ortheir individual aspects and features, can be combined or exchanged withone another without limiting or widening the scope of the describedinvention, whenever such a combination or exchange is meaningful and inthe sense of this invention. Advantages which are described with respectto one embodiment of the present invention are, wherever applicable,also advantageous of other embodiments of the present invention.

Finally, it should again be noted that the devices and methods describedabove in detail are merely example embodiments which can be modified bya person skilled in the art in a wide variety of ways without departingfrom the scope of the invention. Furthermore, the use of the indefinitearticle “a” or “an” does not preclude the possibility that the relevantfeatures can also be present plurally. Similarly, the expression “unit”does not preclude this including a plurality of components which canpossibly also be spatially distributed.

The patent claims of the application are formulation proposals withoutprejudice for obtaining more extensive patent protection. The applicantreserves the right to claim even further combinations of featurespreviously disclosed only in the description and/or drawings.

References back that are used in dependent claims indicate the furtherembodiment of the subject matter of the main claim by way of thefeatures of the respective dependent claim; they should not beunderstood as dispensing with obtaining independent protection of thesubject matter for the combinations of features in the referred-backdependent claims. Furthermore, with regard to interpreting the claims,where a feature is concretized in more specific detail in a subordinateclaim, it should be assumed that such a restriction is not present inthe respective preceding claims.

Since the subject matter of the dependent claims in relation to theprior art on the priority date may form separate and independentinventions, the applicant reserves the right to make them the subjectmatter of independent claims or divisional declarations. They mayfurthermore also contain independent inventions which have aconfiguration that is independent of the subject matters of thepreceding dependent claims.

None of the elements recited in the claims are intended to be ameans-plus-function element within the meaning of 35 U.S.C. § 112(f)unless an element is expressly recited using the phrase “means for” or,in the case of a method claim, using the phrases “operation for” or“step for.”

Example embodiments being thus described, it will be obvious that thesame may be varied in many ways. Such variations are not to be regardedas a departure from the spirit and scope of the present invention, andall such modifications as would be obvious to one skilled in the art areintended to be included within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method for classifying alesion, the computer-implemented method comprising: receiving a firstmedical image of an examination volume, the first medical imagecorresponding to a first examination time; receiving a second medicalimage of the examination volume, the second medical image correspondingto a second examination time, different from the first examination time;determining a first lesion area corresponding to a lesion within thefirst medical image; determining a registration function based on acomparison of the first medical image and the second medical image;determining a second lesion area within the second medical image basedon the registration function and the first lesion area; applying atrained classifying function to first input data and second input data,to generate output data, the first input data being based on the firstlesion area and the second input data being based on the second lesionarea, wherein the trained classifying function is a recurrent neuralnetwork, the first input data and the second input data areindependently used as input data for the recurrent neural network,information from applying the recurrent neural network to the secondinput data is stored within a hidden state and is used as additionalinformation when applying the recurrent neural network to the firstinput data, and the output data of the recurrent neural network dependson the hidden state; and determining a lesion classification based onthe output data.
 2. The computer-implemented method of claim 1, whereinthe recurrent neural network includes at least one LSTM block.
 3. Thecomputer-implemented method of claim 1, further comprising: determiningthe first input data by applying a trained preprocessing function to atleast a part of the first medical image containing the first lesionarea, and determining the second input data by applying the trainedpreprocessing function to at least a part of the second medical imagecontaining the second lesion area.
 4. The computer-implemented method ofclaim 3, wherein the trained preprocessing function is configured bytraining to classify lesions within single medical images.
 5. Thecomputer-implemented method of claim 1, wherein the registrationfunction is a non-rigid registration function.
 6. Thecomputer-implemented method of claim 5, wherein the determining of theregistration function is based on a vector momentum-parameterizedstationary velocity field.
 7. The computer-implemented method of claim1, wherein the first medical image and the second medical image aretwo-dimensional or three-dimensional X-ray based medical images.
 8. Acomputer-implemented method for providing a trained classifyingfunction, the computer-implemented method comprising: receiving a firstmedical training image of a training examination volume, the firstmedical training image corresponding to a first examination time;receiving a second medical training image of the training examinationvolume, the second medical training image corresponding to a secondexamination time, different from the first examination time; determininga first lesion area corresponding to a lesion within the first medicaltraining image; determining a registration function based on acomparison of the first medical training image and the second medicaltraining image; receiving a training classification corresponding to afirst lesion area within the first medical training image; determining asecond lesion area within the second medical training image, based onthe registration function and the first medical training image; applyinga trained classifying function to first training input data and secondtraining input data, to generate training output data, the firsttraining input data being based on the first lesion area and the secondtraining input data being based on the second lesion area, wherein thetrained classifying function is a recurrent neural network, the firsttraining input data and the second training input data are independentlyused as input data for the recurrent neural network, information fromapplying the recurrent neural network to the second training input datais stored within a hidden state and is used as additional informationwhen applying the recurrent neural network to the first training inputdata, and the training output data of the recurrent neural networkdepends on the hidden state; adjusting at least one parameter of thetrained classifying function based on a comparison of the trainingclassification and the training output data; and providing the trainedclassifying function.
 9. A classification system for classifying alesion, the classification system comprising: an interface configured toreceive a first medical image of an examination volume, the firstmedical image corresponding to a first examination time, receive asecond medical image of the examination volume, the second medical imagecorresponding to a second examination time, different from the firstexamination time; and at least one processor configured to determine afirst lesion area corresponding to a lesion within the first medicalimage, determine a registration function based on a comparison of thefirst medical image and the second medical image, determine a secondlesion area within the second medical image based on the registrationfunction and the first lesion area, apply a trained classifying functionto first input data and second input data, to generate output data, thefirst input data being based on the first lesion area and the secondinput data being based on the second lesion area, wherein the trainedclassifying function is a recurrent neural network, the first input dataand the second input data are independently used as input data for therecurrent neural network, information from applying the recurrent neuralnetwork to the second input data is stored within a hidden state and isused as additional information when applying the recurrent neuralnetwork to the first input data, and the output data of the recurrentneural network depends on the hidden state, and determining a lesionclassification based on the output data.
 10. A medical imaging system,comprising: the classification system of claim
 9. 11. A non-transitorycomputer program product storing instructions which, when executed by aclassification system, cause the classification system to carry out thecomputer-implemented method of claim
 1. 12. A non-transitorycomputer-readable medium storing instructions which, when executed by aclassification system, cause the classification system to carry out thecomputer-implemented method of claim
 1. 13. A non-transitorycomputer-readable storage medium, storing a trained classifying functionprovided by the computer-implemented method according to claim 8, foruse in a method for classifying a lesion upon execution by at least oneprocessor.
 14. A non-transitory computer-readable medium storinginstructions which, when executed by at least one processor, cause theat least one processor to carry out the computer-implemented method ofclaim 7.